Repository navigation
[FSDP] FSDP Support Muon and CSA - #79781
Conversation
| out.append((param, grad)) | ||
| return out | ||
|
|
||
| def _muon_sharded_params_grads(self, group): |
There was a problem hiding this comment.
本 PR 新增了 FSDP + Muon 的整套支持,其中按矩阵边界跨 rank 切分(_muon_3d_shard_numel、_muon_sharded_params_grads 以及 broadcast_muon_params 的 muon-shard 分支)配合 ns_per_matrix 的逐矩阵 Newton-Schulz,是一条会直接影响权重更新数值的新分布式路径;但本次改动未新增任何测试,现有 test/collective/fleet/dygraph_group_fsdp.py 也未覆盖 Muon。
作者已在 muon.py 的 ns_per_matrix 文档中说明按 batch 切分会改变结果(bf16 下约 1.2e-2 相对误差),因此该切分路径的正确性与数值一致性需要多卡测试来保证。建议补充代表性的多卡用例:3D Muon 参数按矩阵边界跨 rank 切分,校验切分后最终权重与单卡/复制路径一致;并覆盖对齐回退(shard_align_numel 不满足时退回复制)与 _maybe_force_per_matrix_ns 自动开启 ns_per_matrix 的分支。
There was a problem hiding this comment.
已在最新提交中补充多卡测试 run_muon_shard(test/collective/fleet/dygraph_group_fsdp.py,经 test_dygraph_group_fsdp 以 2 卡启动):对 3D Muon 参数分别走矩阵边界切分路径与 owner 聚合的 dense 切分路径,单步训练后按 bitwise 校验两者最终权重完全一致,并断言 muon_shard_numel == HIDDEN*INTER、is_sharded、muon_owner_rank is None 以及自动开启 ns_per_matrix。该路径的数值一致性已有回归覆盖,此项已解决。
Paddle-Bot Review Board (review完成)
Powered by Nyanpasu claude with Opus 4.8 默认推理级别, please check the suggestions carefully. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #79781 +/- ##
==========================================
Coverage ? 91.86%
==========================================
Files ? 4
Lines ? 369
Branches ? 0
==========================================
Hits ? 339
Misses ? 30
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| if hasattr(fsdp_context, "bind_decay_param_fun"): | ||
| fsdp_context.bind_decay_param_fun(self) | ||
| if hasattr(fsdp_context, "bind_optimizer"): | ||
| fsdp_context.bind_optimizer(self) |
There was a problem hiding this comment.
Muon 重写了 _apply_optimize,这部分逻辑在 Muon 中可能不会被调用?
There was a problem hiding this comment.
感谢,这块是给优化器 save/load 用的,之后会在 muon.py 补上对应调用。
| full = getattr(param, "_muon_full_shape", None) | ||
| if full is None: | ||
| continue | ||
| param.get_tensor()._set_dims(full) |
There was a problem hiding this comment.
L1455 是想要对 local rank Tensor 设置了 global logical shape 吗?
There was a problem hiding this comment.
是的,更新期间参数是本地分片的状态,更新完恢复成完整的 global logical shape。
PR Category
Distributed Strategy
PR Types
New features
Description
FSDP Support Muon and CSA
Check in DSV4 with N4C32_DSV4_PT_seq32K_gbs32_exp256 (FSDP vs PP):
Check in DSV4 with N1C8 (FSDP vs Sharding Stage1):

是否引起精度变化
否